speaking on the question that I think is actually the best-named title of ShmooCon.
Who's got time for that? Looking for advisories and blogs for threat data.
Thanks for that introduction and for the shout out to the title.
Hi everyone, I'm Shimon. I'm Elvis Hovar. And we are both security researchers
working for Accenture Technology Labs. It's the R&D arm of Accenture, if
you've heard of the large consulting firm. And who are we? So both of us are
security researchers. We also have a research team that's focused on natural
language processing. The core mission of Accenture Technology Labs as a whole is
to address essentially challenging problems within certain domains. Elvis
and I focus on security and because of the wealth of expertise there is, we also
do a lot of cross-domain research. So we're essentially working on different
research projects and seeing if we can address any real-world problems. So one
of the things that we have been focused on over the last few months is the
notion of how do you operationalize on threat data. And I will not use the word
threat intelligence because one thing I want to stay away from is at least
giving any false notion that we are trying to tackle the large problem of
threat intelligence. You're still focused on threat data. And what we
want to talk about is the techniques that we are researching to bring to
bear up on from the NLP side, the natural language processing side,
to the threat data extraction within the security domain. Right, so I just want to
create a frame of reference around the problems that we want to talk about. And
when we talk about the challenge space, again, again focus on threat data itself.
Right, when we start talking about threat intelligence, there are a lot of connotations.
Not talking about advisories, we're not talking about indicators, but the actual
data that... How many of you are threat analysts in the room here or spend a lot
of time looking at threat data? Right, it's you folks who are turning the data
into intelligence. What we really want to focus on here is how do we
put the right kind of tools that can make your life easier, make you more
efficient. The human is not going out of the loop. The human will stay in the loop
no matter what we say. So let's try and address challenges that humans are
facing today. What are the kind of desired functionalities, right, as we
started looking at the challenge space? There are some low hanging fruits and
there are some pie in the sky, blue sky ideas. What is the entire spectrum of
those ideas and which are the ones that we picked that we felt were at least
addressable in the near term range? The framework behind how we are
cross-pollinating ideas from natural language processing and information
extraction into the security domain itself. I particularly haven't come
across too much research within the space and I'll be interested in
hearing more about from at the end. If there are any ideas and if you've seen
other research work that's similar. The methodology and results. So this is an
experimental framework. Natural language processing underlying that is machine
learning. It is a continuous iterative process in terms of improving,
leveraging the human intelligence and human expertise to make the machine part
more intelligent. So it is an iterative process. We are going through
this in terms of it is a work in progress, right, so the results that you see are
real results and some of them good, some of them not so hot. So we'll be
pointing out the kind of challenges that we are seeing. Don't want to just talk
about this. This is not supposed to be a pedantic presentation. So what is it
that we have put together that really brings this idea to life and what do we
see going forward in terms of how do you make these improvements and again would
love to hear feedback and opinions around what can be done based on the
kind of things that you're seeing in your day-to-day lives. All right, so as
you as you read through this, I'm not going to drain the slide, this should
resonate with those of you who do spend even a little bit of time trying to
study or trying to look through threat data on a day-to-day basis. To me
it's the 80-20 problem of human analysis. Instead of spending 20% of your
effort to get 80% of the results, it's kind of the other way around and correct
me if I'm wrong, right, how many of you spend a lot of time pre-processing the
advisories, the blogs, the data to actually put it into some kind of a
usable format. Those are essentially pre-processing techniques that can be
easily replicated using computing technology, right. So what we what you
want to try and do in terms of addressing the core issue of trying to
leverage humans to do things that they are actually good at and not wasting
their time on techniques or processes that can be actually automated. Humans
don't scale, right. There are only so many experts and analysts that
we have. Some of the largest threat teams that I've seen are three or four
people. If they are spending most of the time trying to read through documents
then things are bound to fall to the crack. Again, relying on human memory to
be able to detect or remember and co-correlate insights across several
documents is another massive waste of time and energy and effort on
behalf of these analysts. How many of you use spreadsheets to convert data from
PDFs, text documents, blogs and use that as a continual source of repository? No
one? All right, well that's good. It seems like a state of the art has
moved on. So but it is a status quo right now. We are still focused on using
humans where they are not efficient or rather they shouldn't be doing some
of those things. So how do we use computing technology to move the dial
towards that 80-20 rule, right. And as you as you look through this and these are
kind of insights that we came across as we spoke to teams that we support and as
security researchers ourselves is what can we do with thread data that or at
least the problems related to thread data that have been solved in other
domains. National language processing was one of the first things that that came
to mind. Being able to automate the parsing or even semi automate the
parsing of data into some kind of output that is semi-structured. That
itself gets you much closer to being able to use at least some semi
automated techniques for analysis of the data. Being able to share that data, right.
Don't want to still go to the process of picking up the phone or sending an email
which is again in some in some kind of an unstructured format expecting the
person on the receiving end to parse through it and understand what that
format means. So getting to a common understanding is something that we can
get to using national language processing techniques. So as we as we
started talking to the national language processing experts that we have within
with an Accenture Technology Labs it became clear that they're that they have
been working on addressing this problem in other domains. So we figured how do we
bring security expertise, layer that with national language processing and try and
at least address this problem to the point where we're not going to find the
silver bullet here. That was that was never the intent but even if we can
increase the efficiency of human analysts by even 10% I think we are
taking a step in the right direction.
So again I'm not going to drain the slide. It's pretty pretty simple overview of
the framework that we are using to to create this solution for a research
project but there are a couple of things that I do want to focus on. First one is
the information extraction module. I'll be going through exactly why we picked a
specific application within natural language processing which is information
extraction. The other is standardization. The whole one of the objectives that we
set out to achieve was converting this unstructured or semi-structured data into
a common understanding or commonly understood format. Why STICS? Well I'm not
going to go into the justification and rationale behind STICS itself.
It's a standardization effort. Is Sean Barnum here? No. He had a
great talk last year about how standardization efforts help with
sharing insights. The guys at MITRE are doing some really great work. So again
the notion was need to get to a standardized format and we went with
STICS. We also instead of storing the data in just a traditional
relational database we decided to go with a graph database and now as you as
you look at how threat data itself is represented it is multi-dimensional.
There are lots of relationships and being able to maintain these relationships
in a relational database we found out very quickly was was going to be a task
not not worth pursuing. Graph database themselves lend themselves really well
to representing this hyperdimensional relations between between the data
itself. So being able to quickly recall the degrees of separation between
different data between different threat data and the documents that they're
assigned in those were the kind of things that we wanted to represent and
be able to do it quickly. And the third module that I really want to focus on is
the learning part. As I mentioned at start this is a machine learning
technique at the end of the day. There needs to be a way of continually
sending feedback into into the system. So we wanted to give the analysts the
experts the ability to either say yes a certain pattern is good or a certain
pattern is not good. This is this is not a perfect system so we want to use the
effort that analysts are already putting into analyzing this data and help the
system improve over time. NLP 101 just a quick primer. How many of you are
familiar with natural language processing techniques or have done the
same kind of research on it? So there's a few of you here. The primary
purpose is to replicate how humans process language and be able to replicate
the linguistic analysis techniques. Several different applications of
natural language processing. Natural language processing itself is kind of a
catch-all term. Within the field itself there are several very
specific applications and outputs that you get from these different
applications. We focused on information extraction and the idea behind
information extraction is to be able to automate the task of identifying,
collecting, and normalizing the relevant information into a structured output.
So information extraction is more than just keywords. It's more than just
string processing although there are some of those elements involved in
there. But we are not going to the point or to the extent of fully understanding
the text the way humans have the capability of doing. So these
applications are very domain specific. Information extraction works well if you
can represent the human expertise within a specific domain the right way. So that
was where a lot of our time and effort was spent. Great school anyone?
This is just a quick review of the linguistic analysis that we go through.
Humans do this really easily, very intuitively. Trying to have a
machine do this is an extremely difficult challenge. It is not a
trivial challenge. So as we go through the increasing levels of
analysis, we looked at doing morphological analysis which is breaking
the words or a complex word into its basic form. So if you take for example
the sentence, the three buffer overflow vulnerabilities could allow remote
code execution. So when you look at the word overflow,
break it down into over and flow. Try and get to the basic units. Understanding the
syntax, so are the words correctly placed and what kind of meaning are you getting
out of it? And the third level of analysis, the semantic analysis itself,
which is really understanding what does a word mean within the context of the
sentence. So if you look at the word buffer, there are several meanings. There's
noun form, the verb form. Within the context of computation, it means a
storage, right? It's storage. So being able to get to the right meaning of the word
is extremely important. So as we started looking at and working with our
natural language processing experts, these were the kind of things that we
really focused on. So just a high level summary of approach. Now here's where we
really dive into the kind of framework that we set up for extracting the data
and also we'll talk about the different algorithms that we used for
analyzing the data. So it was a six-step process. The first step, just break the
entire text into its sentences, right? So sentence segmentation. The second step,
which was a core part of our work, was to generate patterns related to each of the
STICs data constructs. So when you look at IOCs, TTPs, course of action, we
generated base patterns that would be used for the analysis in later stages.
Step three was stemming. This is again an NLP technique to reduce the
complexity in the actual language itself. So stemming essentially boils or
distills different forms of the word to its basic form. Think hacker, hackers,
hacking, they all boil down to hack. So that's the idea behind stemming,
reducing the complexity. Now step four is where you take each sentence and
calculate the similarity score by comparing the sentence against all the
patterns that we had already generated in step two, right? So if you
take a sentence, compare it against all the patterns in course of action. Take a
sentence, compare it against all the patterns within TTPs, IOCs, and so on.
Once you went through that entire process, we'd find the best match in
terms of the similarity score, the pattern set that was or the patterns
that were actually matched against that sentence with the highest score. And then
we repeated steps four and five for each STIX data construct. So that was
the basic idea behind this entire process.
So for the pattern generation itself, this is a very crucial
component of information extraction. Garbage in, garbage out. Very true
for information extraction. So it was important that we started out with
at least the right base patterns. We use a couple of different approaches
for this. The first one was a supervised approach. We actually, Elvis,
manually annotated the pattern list that needed to be learned. This is essentially
a classification challenge. So you have a corpus of documents, manually annotate
them, feed them through the classifier, and let the classifier learn, right? So
we call it a supervised approach. And then we also use a semi-supervised
approach. This was used to boost the number of patterns, the base patterns
that we could generate. The supervised approach is fairly tedious. It does
require a lot of time and effort. So we wanted to boost the actual pattern list.
And we used the semi-supervised approach. The idea here is you create a set of seed
patterns and you feed it documents that are not annotated. And because they're not
annotated, this goes through a bootstrapping process where it keeps learning
over time. There is human intervention required. If you just allow it to learn
on its own and if it keeps learning garbage, then you'll end up with garbage
at the end. So that was another part of our effort was to find the right kind of
base patterns. And let me quickly go to the next slide so Elvis can then really
show you what this looks like in real life. So we went through several
iterations of training and testing. We used documents and advisories from MSISAC,
FSISAC, and ICSRT, ran it through the information extraction process, matched
it against all the learned patterns that we had created, right? So the patterns
that we created in the previous step, matched it against those, and we
calculated three different similarity scores, lexical, semantic, and contextual
similarity. Now each of these have their own advantages and disadvantages.
So the lexical similarity score works well when you're looking for co-occurring
set of words. So think Siemens and Symantec or any other ICS domain product.
There are a certain set of co-occurring words that you can identify. The
semantic similarity score tried to detect a concept through a word or a
phrase, and then the contextual similarity score analyzed a certain window of words
around a known pattern to try and understand the context. As Elvis goes to
the demo, he'll talk about why we generated the three different scores and
how they were useful in being able to map to the right STIX construct. So
aggregate the score, look at the result, and if the results good, then make sure
that the pattern is in the pattern list. If not, add it to it. If the score is bad,
try and understand what went wrong, finding the parameters. This is your
traditional training and testing process that you use for machine learning. So
enough slides. Just want to make sure that everyone's awake. Threat intelligence.
Any schmooble? No. All right. So Elvis.
Can I drop the mic when I'm done? Hi everyone. My name is Elvis Hovor. I have
been with Accenture Technology Labs about three years. I started out of grad
school. I lead some of the development work in threat intelligence down there.
Okay, so let's start the demo if I can find it.
I don't have anything higher than 1440. That's the best I have.
Anyway, is it showing any better now? What can I do? Help here. Reduce it. Zoom out.
Okay. That didn't help.
That good? Thank you. Okay. So I think like Shimon said, initially we started with
trying to think about, you know, how to help analysts really prioritize it.
Advice we study get on a daily basis. I'm sure that a bunch of analysts here you
get documents on the regular that you need to read through every morning. Try
and figure out which one to read first is a task on its own. You know, which one
has the most important information or relevant information to you is also
another task that you have to go through. Wasting your time trying to figure out
which document to read and all of that. Shimon went through a lot of that. What we
wanted to do with our research project was initially started with just trying
to prioritize these documents. But as time went on we decided to wrap a UI
around that to make it easy and more usable for anyone that wanted to use a
tool. So we moved away from just the research portion of it to try and put a
UI on top of it. What you see here is, you know, on a regular day you come in and
load up bulk documents basically. It is placed on this page and it
shows you scores on which one we think is, well, you have prioritized the most.
Once again this is a research idea we give a scores because we thought that
those were a good way to score the documents that we have. But, you know,
based on your own preferences you can score it differently and it will show
differently. But you look here and then you see a set of documents and these
are the ones that are prioritized for you to read first and second and third. It
came in on different days. We have a set of data that spans maybe three days. We
wish we had more but it spans about three or four days. That's about how much
we have. I would try and go through the process with you loading up how we are
calculating the scores and what kind of information we are getting out of it.
The data elements we are able to extract out of each document. What extraction
method works better for us and what we realized would work to extract certain
six elements or constructs better than other ones. And so as I go through I'll
explain that more. So come here and upload a data. First I would upload a file that
has, I guess, the patterns which trained or according to that file structure. So
extracts a lot better if you trained your your pattern set against a specific
document and have trained your modules over time to be able to extract that
data more. So you would see that one works very well and the other works
somewhat okay. So let me try and upload.
Okay this is a file I can upload.
Oh and we were trying to, we're trying to make it a batch file because it takes
sometimes it takes just a little bit of time to run a single file. So we're
thinking if someone has multiple files that they want to run you want to be
able to make sure that you can make it a bad job for the first one to process
and the second one to come in. But for some reason Java wouldn't allow, well we
haven't figured out how to have the computer run the Java command on its own
because it's kind of like sandbox and would allow us unless we enter the
command. So I have to enter it here but that gives you an opportunity to see
what's going on in the background. How the you know how it's comparing to the
patterns like Shimon showed earlier on.
So that's the extraction that's going on right now. So if I get back to my page
here and I go to my prioritized list at the end of a list table you should be
able to see the file that we just entered. We shorted out a little bit but
what eventually happens is that when you run it through and the natural language
processing module is running in the back and they should tell you that it's being
processed and when it's done when you refresh your page you would see it shows
an output that says that it's been processed and so you can go ahead and
start looking at the file. So if you have a bunch of files this is going to be a
batch process that keeps going through and taking the files through. I have to
shift this a little bit just so I can get to that. So that's it. The file is
uploaded it's running right now. I'm sure it's done if I highlight it it should
move. Please pay attention to the file name I think it's USCIC cert.
Okay yeah thank you.
So now we that means that it found enough information in there to think and
score it to take it to the third in the in your stack of documents that you have.
I'll tell you a little bit about the scoring as we go through and why the
scoring is that way. It's not perfect yet but as time goes on I think we would
tweak the scoring a lot more to get some of the information that we want out of
it. So the document will show on the right hand side and this elements
extracted would be on the right hand side. So you see your COAs all the things
that it was able to extract out of the document you would see from here. So for
COAs we can see that extracted a bunch of things out. The score you see here is
1.0. 1.0 because anything that comes out of contextual similarity extraction if
we use that method gives us a binary number it's either 0 or 1. It was able to
extract something it wasn't. Semantic and lexical are a little more different in
that the scores actually work and you can tweak them very well. So that is one
part that kind of skews our score a little bit but what ends up happening is
that if you still don't find anything for or if you find contextual which is
one we decided to average it out for a document. So if lexical and semantic
similarity methods didn't find anything you would still average it out by three
and it ends up reducing the value of the score. So we were able to find some COAs.
It is it gets because of the patterns that we have sometimes it can extract
certain information that is not necessarily accurate or extracts before
it's supposed to extract the full sentence. We because of this we wanted to
build in that like Shimon said earlier on a learning module to make sure that
we are correcting this automatically or you know then the module in the back end
is learning to update itself and delete the patterns that don't work and all of
that. So we put the Facebook style thumbs down there just to you know delete
anything that we believe is wrong. So now it's taking all of the how-to's out
because it was extracting how-to's and what we built inside that again is the
ability to delete this pattern whatever pattern is pulling out the how-to if it
reaches a certain threshold if you get it gets a certain number of
and thumbs down. So that way the module is learning and as time goes on we want
to be able to build functionality for analysts to be able to you know
highlight a text or highlight a pattern that he thinks the natural language
processing module is missing and then it's going to add on to a set of a
documents that we have. You can see here TTPs that IOCs in here and they pick up
pretty good ones if the document is really known and and the patterns were
tested with that document. I would go in and pick a regular document from cert
and upload that and see what the difference is you would see that some of
them would not be as great as what we have and with the documents that have
been tested already. So what, first one?
Goodness. Okay, TXT. Upload it again.
Once again we should be able to see it being processed and when we highlighted
when it's done we should be able to get into the file and check that out. So
whilst that is running and these are some of the things that these are some
of the findings that we we saw as we the research project went on. We're hoping
that we could use one of the similarity methods, the lexical contextual or
semantic similarity methods to extract the information that we needed but we
we realized that each one has its own advantage so it would it would not be
wise for us to end up to just choose one and leave the others because we we saw
that maybe for contextual similarity was able to pull things like thread actors
the things that have very specific elements that need to be pulled out like
a word or you know a very small phrase a phrase that isn't too long and then for
things like TTPs that sometimes can span multiple lines we saw that using things
like semantic or lexical similarity methods will pull those out better for
us so we ended up using an aggregate of all of these three to be able to get
this the information that we needed. So I believe it is done and we can check that out.
Э
That's that's it.
So out of this document, it was able to pull a bunch of exploit targets for us.
As you can see, it grabs, sometimes it can grab the sentence that you want and not necessarily
give you the element that you want.
It might grab it before, it might grab it after.
It's just, I guess, the patterns and then having to get the right pattern match for
each document.
The thing with natural language processing, and I think it's natural language processing
as a whole, it's still being developed to get to that fine-grained point where it can
pull very specific information.
And so as much as we were trying to get it, and this is also, I guess, a project that
we are still building on and trying to get to the next step, we do have a few things
that need to be tweaked here and there, but it is extracting some information.
So if you, as an analyst, can get to move away from doing all of these things and having
just a document moving up and telling you that I have this amount of information on
there that is related to threat information, I think you should look at that and see if
that applies to you.
It's going to make your job a little more easier.
So after we were done with putting this information in a more structured format, which is the
sticks that we have here, and I think it's a caveat to putting it in a structured format
in sticks, and I'll put that in our challenges at the end, but we wanted to do the higher
order analysis that I think Shimon spoke about early on.
I mean, if we have the information in a more structured format, there's so much we can
do with it.
So we decided to take some examples and do some D3 visualizations on top of that, of
the data that we have to see what we can pull out and what we can do with it.
Like we said, it's just examples.
We had a graph database.
We were pulling out the information out and trying to see what we could relate together
and the kind of information that we can pull out of it now that it's more structured.
So we decided to use this tree map.
This is one of the first ones that we have.
To show the information that we have in there, and thanks to Sean, D3 guy, he helped us a
lot on this.
So we decided that if we could use a tree map to see what information is contained in
the documents, then you can have a general idea.
If you limit the documents that you have to maybe a one week set of documents, and then
you want to see what has shown up a lot in your advice that you've collected.
I think one thing that I've forgotten to say is that as analysts, or as the threat data
team, or the threat analyst team, you collect documents that are usually related to your
organization.
So some of this information can get very specific to your organization because you're collecting
data that's around the kind of operations that you guys run in your organizations.
So here you can come here and just look for maybe threat actors and see in the last week
which threat actors have shown up a lot in the documents that I have, or in the last
day, maybe last month, which one is coming up a lot and why should I be worried about
that.
And then you can do the same for COAs, go into your COAs and see what kind of COAs are
out there.
And the COAs are, the word sentences, or very long sentences, trying to display that in
D3 was a little bit of a hassle too.
We're still trying to figure that out, like we said.
So we want to be able to give the user the ability to be able to see what the whole COA
is.
When I say COA here, I mean causes of action.
I'm sorry.
I'm using a lot of acronyms, am I not?
Anyway.
The other visualization we decided to use was historical analysis.
Here we wanted to either give the analyst the ability to widen the net or tighten the
net some more.
You have documents, and I think one of the problems that we're talking about earlier
on is that as an analyst, you, or as a human trying to analyze these documents, you seem
to may have seen some element in there, maybe a threat actor or a TTP that you were concerned
about, but it was way back, let's say four weeks back or a month back, and you don't
remember which document it was anymore.
You never did anything about it, but all of a sudden you see that picking up again and
you want to be able to find those documents that are related to that document that you
just saw, but there's no way you can remember it.
You don't even know sometimes if you ever came across that.
We wanted to show that information here and make sure that we give you a historical view
of the kind of documents that you have in your repository or your database.
What you see here is a list of the documents that we have in here.
It's a limited list that we have.
Anytime you mouse over a document, let's say you just put a new document in, it shows up.
Anytime you mouse over it, you see the documents that are connected to that document in any
way.
If it's a set of threat actors that are connected, it's going to show you a different color.
If it's a set of IOCs that are connected, it's going to show you a different color.
It just gives you an idea of how your documents are connected together and which ones you
should be paying attention to if you ever want to go back and read or look at some of
information that you have in there.
Like we said, this is all going into our Neo4j database.
So Neo4j has a frontend query.
You can go in there and query the data that you want and do it whichever way you want.
This is D3.
It's all open source.
You can decide to use whichever visualization you want to use on top of this data to pull
whatever you feel is most necessary for you.
These were just examples we wanted to show.
And so some of the challenges that I guess we had going into this and we realized were
going to be a hindrance for us is that, you know, stakes.
We wanted to put it in stakes.
We said stakes.
We wanted to make sure that everything is in stakes.
It's more structured.
But stakes is very expensive.
And the data or the elements that you need to extract into stakes can get very, very,
very granular and you need to make sure that you capture that out of a sentence or you
capture that without missing the meaning of that in the sentence.
And it became very difficult.
We tried with natural language techniques to get granular and we saw that we were missing
a lot of information that we needed.
So we decided to expand it a little more and go for the broader stakes constructs, which
is IOCs and, you know, IOCs, threat actors, things of that sort.
It's broader.
That way we can capture a lot more.
You might have to capture it in a sentence instead of capturing the element, but it still
gives you a better idea of what's in there.
And as time goes on and as this is developed and as people, you know, put a lot more effort
into it, we believe that it's going to get to that point where the natural language processing
engine will be able to go directly and look for things like affected systems and just
pull that affected system out and put it in a very, very specific place that a machine
can easily use.
Do you have anything?
Okay.
Yeah, so that was one of it and also I think this is one of the most basic NLP problems
around having one where that means something else in a different sentence.
So it's difficult.
As humans, we are easily able to kind of make those out, but for a computer, it's a lot
more difficult for the computer to make out that, you know, maybe, what?
Give me an example.
Maybe attacker, right?
In one sentence means that that's the person that actually committed a crime.
It's a threat actor.
And it's simple form, attack simple form is actually attack, right?
And you know, for the computer, it sees it, it tries to break it down into all of its
simplest forms for natural language processing and all of a sudden that attack just means
it's trying to explain something else in the middle of a sentence somewhere in TTP.
It ends up pulling the same attack just like it's trying to pull attacker.
So things like that made it very difficult.
These are underlining NLP problems and I think that as we go forward and we put a little
more effort into it as a community, it's going to improve and this is going to help in some
of the work that we're trying to do.
All right.
Let me just take it home.
So yeah, in terms of what are we trying to do moving forward, right?
Like we talked a little bit about some of the challenges that we face, some of the results
that work, some of the patterns that work.
What we, one of the first things that we want to do is extend the functionality of tagging
new patterns from the screen saw, right?
So that's really put the power of the tool in the hands of the analyst.
There's a lot of fine-tuning to be done with the weighting parameters.
We are, as we go forward, one of the things that we found when we were doing the analysis,
threat actors and I was talking about threat actors, TTPs, were getting really accurate
results using contextual similarity, but then there are highly specific elements within indicators
of compromise, within observables that are better suited for lexical and semantic analysis.
So we are looking at those components as we go through the iterative process of making
this solution and the underlying engine better.
Expand the pattern sets for different types of documents.
Right now we are heavily focused on USERT, ICSERT, and MS-ISAC.
Those documents tend to have a lot more vulnerability, courses of action.
Sometimes they might have threat actor and campaign names.
Unusual.
So we want to expand the actual corpus to include that spectrum of information.
And one of the next things that the National Language Processing team is really going to
focus on is being able to incorporate named entity recognition.
So being able to identify references to known names and entities in the text themselves.
This can be really useful if people do follow a common standard of naming campaigns and
threat actors.
These can be used for named entity recognition techniques.
So everything that we've used here is based off open source technology.
And if you do want to tinker, and I believe quite a lot of you here are hobbyists, if
you want to build your own stack, where do you go about putting this together?
There aren't a lot of NLP people here.
We had to go about trying to rely on the knowledge of our team to really understand how do you
get to put this whole stack together.
So Apache Open NLP, definitely look that up.
Excellent resource of both implemented algorithms and documentation.
They have all the underlying technologies that you need for parsing the sentences, applying
stemming, doing any kind of part of speech tagging, any kind of basic parsing.
They have a lot of great implemented algorithms already out there.
The Stanford National Language Processing Group, again, a great wealth of information
over there.
They also have released a lot of their algorithms on their website.
And WordNet, it's essentially a lexical database of English.
So whenever you're trying to do similarity scores between words that could have the same
meaning or could be related, WordNet is the world's largest lexical database.
So definitely that's a core component of what we are using.
And then the TextRank algorithm, again, this is a core part of how we are doing the lexical
similarity matching or how we intend on implementing lexical similarity matching.
All of this is open source.
It's available out there for you to go download.
And one of the things that we want to do over the next, in the near future, is we'll be
setting up, we're looking to set up a GitHub repository and at least start a flow of information
of our learnings into the open community.
So yeah, with that, I think we still have a few minutes for questions.
I'm sure you guys have a few.
And thanks for your time.
And if you do want to follow us, you can follow me on Twitter.
Hit me up with any questions.
Any if you want to take anything offline, we'll be happy to start a conversation.
But with that, with whatever little time that we have left, we'd be happy to take any questions.
Thank you.